<html>
<head>
<title>Automatic Grading System - Project 2</title>
</head>
<body style="font-family:'arial'">
<h1>Project 2</h1>

<table width='600' border = '1'>
<tr >
<th align='left' width='200'><a href='../proj1a/main.html'>PROJECT 1A</a></th>
<th align='left' width='200'><a href='../proj1b/main.html'>PROJECT 1B</a></th>
<th align='left' width='200'><a href='../index.html'>RETURN TO MAIN</a></th>
</tr>
<table>

<hr>
Source Code is available at: <a href='https://bitbucket.org/chucheng/cs143_autograding'>https://bitbucket.org/chucheng/cs143_autograding</a>. Please consult ChuCheng Hsieh for access (chucheng@ucla.edu)   

<hr>

<b>OVERVIEW</b><br>
In Project 2, students need to implement B+ tree index for an existed SQL engine. Even though students submit their work separately in 4 phases, the project is graded altogether when all the work are submitted. There are about 25 testcases in the module which can easily configured by graders for testing purposes. Grader needs to ensure the testcases and solutions followed the valid format so that system can process them correctly. This page contains the instructions for running the grading script and also some important notes of this module.
<br>
<hr>
<div style="width:600px;height:140px;border:2px dotted black;">
<b><u>BEFORE GRADING:</u></b>
<ul>
<li>Please install the same environment prepared by John Cho. Instruction can be found at <a href='../index.html'>main page</a>.</li>
<li>Please set up internet connection for the virtual machine according to the instruction above.</li>
<li>Please make sure your student submission folder follow the required <a href='#structure'>structure</a>.</li>
</ul>
</div>

<br>

<div style="width:725px;height:2350px;border:2px dotted black;">
<b><u>HOW TO RUN THE PROGRAM:</u></b>
<h5 id='structure'>STEP1:</h5>
<p>Check the submission folder format.
<ol>
<li>The submission directory will contain 4 sub directories with submission: a, b, c,and d</li>
<li>There should be students' submissions within each sub directory. Also, there should be "<b style='color:red'>submission.csv</b>" in each sub-directories, which is the log for student submissions. This file is crucial in the grading process to organize student's files into single compilable set.</li>
<li>The individual student's directory will contain their work for that sub-part.</li>
</ol>
</p>
<div style="width:720px;height:314px;border:2px solid black;"><img src='./p2/step1.jpg' alt='This screenshot shows the submission folder structure'></div>
<br>


<h5>STEP2:</h5>
Configure the testcases if there are any changes according to the grading guideline provided by teaching assistant. Please find the <a href='#testcases'>file format</a> below.<br> 

<h5>STEP3:</h5>
Run the python script with your arguments. Please find the available <a href='#cfg'>arguments</a> below.<br> 
<div style="font-size:10pt"><b><i>- Type "./main.py -s [directory to be graded]" to execute the program.</b></i></div>
<div style="font-size:10pt"><b><i>- "*** stack smashing detected ***" error in the screenshot is because of erroneous returned set from student's project, which causes the stack overflow. The warning does not has any impact on grading system.</b></i></div>

<div style="width:720px;height:256px;border:2px solid black;"><img src='./p2/step3.jpg' alt='This screenshot shows an example for how to run the script'></div>

<h5>STEP4: <font color='red'>GRADING COMPLETED!!</font></h5>
Please find the grading result in "<b>RESULT/gradebook.txt</b>". This folder is generated in the grading process, and the gradebook file is in the tab delimited form.
<div style="width:720px;height:388px;border:2px solid black;"><img src='./p2/gradebook.jpg' alt='This screenshot shows the location of gradebook file'><br>
</div>
<br>
In the <b>COMMENTS</b> field of the gradebook, the system uses the ERROR CODE to distinguish the mistake in each testcase.<br>
<div style="width:350px;height:125px;border:1px dotted black;">
<b>LD</b>-- Table load failed<br>
<b>Err[num]</b>-- Query failed with return code [num]<br>
<b>TO</b>-- Time Out Error<br>
<b>PR</b>-- Wrong page read <br>
<b>PRP</b>-- Partially correct page read<br>
<b>PT</b>-- Process time is too long<br>
<b>WA</b>-- Wrong answer set<br>                    
</div>
<br><br><br>
<b id='cfg'><u>CONFIGURATIONS:</u></b>
<br>

The configurations of this program can be modified in <b>config.py</b>. Moreover, some grading parameters are configured upon execution.

Before running the program, grader can provide the following configs:
<ul>
<li> Student submission directory [-s].<br> <div style="font-size:10pt"><b><i>[Default: The module will use "submissions" in the current directory for grading]</i></b></div></li>
<li> Timeout value in SECOND [-t].<br> <div style="font-size:10pt"><b><i>[Default: 15 second]</i></b></div></li>
<li> Grade specific student's project only [-n].</li>
</ul>
<u style="font-size:10pt"><b>The following screenshot is the usage information of the grading script:</b></u>
<div style="width:720px;height:256px;border:2px solid black;"><img src='./p2/help.jpg' alt='This screenshot shows the grading configuration usage page'></div>
<br>

The following parameters are also configurable in this module; however, they require user to modify the config.py manually.
<ol>
<li>The name and location of the configuration files (e.g gradebook output file, testcase file, and solution file).
<font color='red'>Note: the test case script and solution need to be provided in these file explicitly.</font>
</li>
<li>Points for each testing criteria.</li>
<li>Name of sub-directories.</li>
</ol>
<br>
</div>

<br>
<u><b>IMPORTANT NOTES</b></u><br>
<div id='testcases'>1. Testcase & Solution Format: </div>
<div style="font-size:10pt">Based on the grading guideline, the grading system uses the configuration files to store the testcases and solutions. 
These files are required to meet the following format.
<br><br>
<b><u>Testcase file data format: </u></b><br>
<b>File name</b>: testcases.txt (configurable) <br>
<b>File format</b>: <br>
[Testcase Number]||[Index]||[Type]||[Category]||[Query to run]||[Secondary query]
<ul>
<li>Testcase Number : Integer ( 0 based )</li>
<li>Index : Index or Non-index </li><br>
<b style="font-size:8pt">All NON-INDEX testcases will be tested in the second half of the grading process.</b>
<li>Type : LOAD or TEST -- TEST for SELECT statement</li>
<li>Category : xsmall, small, medium, large, xlarge </li>
<li>Query to run : SQL query of the testcase</li>
<li>Secondary query : Some testcases have secondary queries. If the secondary return the same result set as first query, they can still get partial points.</li>
</ul>
</div><br>
<b style="font-size:10pt"><u>Testcase file (sample)</u></b>
<div style="width:800px;height:100px;border:2px dotted black;font-size:10pt;font-family:courier">
14||I||TEST||large||SELECT * FROM large WHERE key <> 88 AND value = 'Alaska'||SELECT * FROM large<br>
17||I||LOAD||xlarge||LOAD xlarge FROM 'xlarge.del' WITH INDEX||<br>
18||I||TEST||xlarge||SELECT COUNT(*) FROM xlarge WHERE key < 3000||<br>
19||I||TEST||xlarge||SELECT * FROM xlarge WHERE key > 40471000 AND value = 'Superman'||<br>
0||NI||TEST||xsmall||SELECT count(*) FROM xsmall WHERE key > 2000||<br>
1||NI||TEST||small||SELECT value FROM small WHERE key < 50||
</div>
<hr>
<div style="font-size:10pt">All testcases are evaluated in three aspect: Accuracy, Efficiency, and Performance. Accuracy is measured by the returning set or data-load result. Efficiency is based on the number of page read. The total processing time is the metric of performance. To avoid multiple files, the processing time and pages read metric is stored in one file, and the answer set of SELECT query is stored in individual file with specific file name.
<br><br>
<b><u>Solution file format for page read and processing time:</u></b><br>
<b>File name</b>: testcases_sol.txt (configurable) <br>
<b>File format</b>: <br>
[Testcase Number]||[Index]||[PAGE READ MIN]||[PAGE READ MAX]||[PAGE READ MIN (Partial pt)]||[PAGE READ MAX (Partial pt)]||[MIN TIME OUT]                                            
<ul>
<li>Testcase Number : Integer (It will match the number in testcase file)</li>
<li>Index : Index or Non-index </li><br>
<li>PAGE READ MIN & PAGE READ MAX : The expected range of the page read</li>
<li>PAGE READ MIN (Partial pt) & PAGE READ MAX (Partial pt) : If the range is provided, student can get partial point if the page read falls in this range</li>
<li>MIN TIME OUT : the minimal process time without getting points deducted.</li>
</ul>
</div>

<b style="font-size:10pt"><u>Solution file for page read and processing time (sample)</u></b>
<div style="width:200px;height:75px;border:2px dotted black;font-size:10pt;font-family:courier">
1||I||1||3||||||0.01<br>
2||I||1||3||||||0.01<br>
4||I||6||8||9||16||0.01<br>
5||I||3||6||||||0.01<br>
1||NI||6||8||||||0.01
</div>
<hr>
<div style="font-size:10pt">
<b><u>Solution filename format for testcase's answer set:</u></b><br>
The answer set for each SELECT testcase is stored separately. As a result, it is required to have unique filename based on the testcase's category. <br>
Filename format: [Testcase Number]_[Index]_[Category]_sol.txt<br>
<ul>
<li>Testcase Number : Integer (It will match the number in testcase file)</li>
<li>Index : Index or Non-index </li><br>
<li>Category : xsmall, small, medium, large, xlarge </li>
</ul>
<div style="width:545px;height:65px;border:2px solid black;"><img src='./p2/testcase_fn.jpg' alt='This screenshot shows the sample of testcase filename'></div>
<br><br>
<b style="font-size:10pt;color:red">NOTE</b>: Solution for LOAD statement will be configured in <b>load_timeout.txt</b>. The solution is only for INDEX loading since the grading module does not grade on non-index load. Please find the format below.
<div style="width:90px;height:75px;border:2px dotted black;font-size:10pt;font-family:courier">
xsmall||3</br>
small||3</br>
medium||3</br>
large||3</br>
xlarge||5
</div>
</div>

<br>
2. The process log has the information of submissions count, error if any, grading note, and total grading time of the grading script ... etc.
<div style="width:720px;height:340px;border:2px solid black;"><img src='./p2/log.jpg' alt='This screenshot shows the sample of log file'></div>

</body>
</html>